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Summary of Review 

According to a new study of Milwaukee public schools, student achievement has benefited 
from voucher-based school competition. A novel method, using geocoding, was proposed for 
measuring the degree of competition within the city of Milwaukee and, in turn, for determining 
whether such competition has increased or decreased the achievement of public school students. 
Though a more traditional measurement of competition was eventually used in lieu of geocoding, 
the authors of the study determined that the overall effect of competition on student outcomes 
was positive over the seven-year span for which data were available. Specifically, it was argued 
that increased school choice improves the academic performance of students in traditional public 
schools who are voucher eligible by means of system-wide competitive pressures. Based on a re- 
view of several key issues — including statistical modeling and control, effect size interpretation, 
the role of explanation in causal inference, and the validity of reported conclusions — the practical 
effect of competition through vouchers appears to be small, if not negligible. It is also suggested 
that a number of methodological issues would benefit from greater clarity. 
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Review 



I. Introduction 

The School Choice Demonstration Project, a 
research center self-described as “devoted to 
the non-partisan study of the effects of school 
choice policy,” recently released The Effect 
of Milwaukee’s Parent Choice Program 
[MPCP] on Student Achievement in Milwau- 
kee Public Schools, a study by Jay P. Greene 
and Ryan H. Marsh. 1 This report (referred to 
hereafter as EMPCP) describes the design 
and execution of a statistical analysis of stu- 
dent and school information obtained from 
1999 to 2006. A novel method, using geo- 
coding, is offered for measuring competition 
within the city of Milwaukee and for deter- 
mining whether there is an association be- 
tween this measure of competition and public 
school student achievement. 2 

Based on their analyses, Greene and Marsh 
conclude that “students in Milwaukee fare 
better academically when they have more 
free private options through the voucher 
program.” They further surmise that “the 
existence over the last two decades of 
MPCP” may have improved student scores 
substantially,” with a mild warning against 
extrapolating the findings beyond the 1999- 
2006 period for which data were collected. 

II. Findings and Conclusions 
Of The Report 

Notwithstanding the concerns described 
later in this review regarding methods and 
interpretation, the report is to be com- 
mended for a thorough and fair presentation 
of results. Reports such as this should make 
public the methods and statistical models 
used in a study and should link empirical 
results and interpretations in an explicit 
manner, and this is done well here. 



The main statistical results are given in the 
report’s Table A, in the section labeled “No 
Controls” under the column labeled “Whole 
City.” This data set contains assessment 
information merged from two achievement 
tests (Wisconsin Knowledge Concepts Ex- 
amination and the Terra Nova Achievement 
Examination). Importantly, the records in 
the analytic sample are for public school, not 
voucher school, students. The study there- 
fore focuses on the potential benefits of 
competition for public school students only. 

In Table A, statistically significant and posi- 
tive regression coefficients (RC, for short) 
were obtained for Language Arts, Mathe- 
matics, and Reading. To help readers under- 
stand this somewhat arcane information, the 
authors provide a more accessible version of 
quantitative results in Table F, in terms of 
effect size? In Language Arts, for example, 
an effect size of ES = .10 is shown in the 
first row of Table F. This is intended to rep- 
resent the effect of voucher competition over 
a seven-year period (the span of the data 
collection); the report does not annualize the 
reported effect sizes, a point given more 
attention below. 

In the first four columns of Table A, results 
are provided for localized competition (us- 
ing the geocoded variables) — especially the 
column indicating competition within about 
1 mile of the public school. No statistically 
significant results are found for the model 
that controls for grade and year. 

III. The Report’s Use of 
Research Literature 

In the section “Prior Research,” a number of 
studies are cited that provide a helpful per- 
spective on what is currently known regard- 
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ing the effects of school competition on stu- 
dent achievement. A number of studies hav- 
ing conflicting results (e.g., Carnoy et al. 
and Hoxby 4 ) are briefly summarized, and 
the authors observe that evidence from stud- 
ies conducted in Florida “gives us some rea- 
son to believe that the threat of voucher 
competition spurs significant improvement 
in traditional public schools” (p. 4). 

The report includes the following quote from 
a comprehensive review of the literature by 
Belfield and Levin, who synthesized 41 em- 
pirical studies of competition, including stud- 
ies of school achievement due to vouchers, 
charter schools, or other public schools: 

A sizable majority [of the 41 studies] 
report beneficial effects of competi- 
tion, and many report statistically 
significant correlations. For each 
study, the effect size of an increase 
of competition by one standard de- 
viation [SD] is reported. The positive 
gains from competition are modest 
with respect to realistic changes in 
levels of competition. The review 
also notes several methodological 
challenges and recommends caution 
in reasoning from point estimates to 
public policy. 5 

The information in this quote suggests the 
following basic method for determining the 
effect of competition. First, an independent 
(or predictor) variable is required that quan- 
tifies competition. Belfield and Levin pro- 
vided a brief review of the literature on 
measuring the degree of competition. Sec- 
ond, an outcome variable is needed, and this 
is typically student achievement. Finally, a 
statistical model is used to link independent 
and outcome variables, thereby obtaining 
quantitative measures of impact. 

Given the novel method of geocoding em- 



ployed in EMPCP (as explained below), 
some conceptual connection with the previ- 
ous literature on measuring competition 
would have been helpful. 6 It also would 
have been helpful if the quantitative findings 
of Belfield and Levin for student outcomes 
had been mentioned; roughly speaking, the 
latter found an average benefit of ES = .10. 
This provides some context for interpreting 
the size of competition effects, though I 
could not determine whether Belfield and 
Levin computed annualized competition 
gains (I suspect not). Because different 
competition studies have different time hori- 
zons, it is difficult to compare benefits with- 
out some kind of time standardization. 

IV. Review of the Report’s 
Methods 

There are two important aspects of the 
method employed in EMPCP to obtain sta- 
tistical estimates: (1) how analytic variables 
are constructed, and (2) model specification, 
in the sense of what variables should, and 
should not, be in the model. 

Constructing Variables 

Although a novel method involving geocod- 
ing was used to construct analytic variables, 
the key positive findings ultimately set forth 
in the report apply to the city as a whole (all 
of MPS), rather than to schools within 
smaller, competitive geographic zones. The 
geocoding was used because of an initial 
belief that competition effects would be felt 
more strongly by a public school when 
voucher-accepting private schools are located 
nearby. The empirical analyses, however, did 
not confirm this conjecture. 

In the end, the main independent variable 
was the interaction term FRL*N, where FRL 
is the free-lunch status of students (repre- 
senting a proxy for eligibility for a voucher). 
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The variable N is the total number of 
voucher-accepting schools in Milwaukee 
that served the grade level of the student. 
This interaction ( FRL*N) can then be de- 
scribed as measuring the combined influ- 
ences of voucher-eligibility and system-wide 
competition of schools (rather than competi- 
tion resulting from nearby schools), or as 
stated in the report (p. 1), “effects on the 
whole school system, not its effects on par- 
ticipating students.” Note that the coefficient 
estimated for FRL*N measures a competi- 
tion effect limited to public school students 
who are voucher-eligible. 

Statistical Model Specification 

A simple OLS (ordinary least squares) re- 
gression model was run, incorporating 
voucher, control, and student fixed effects. 
In addition to the no-controls model, the 
report presents models using “year,” “grade” 
and “year and grade” as controls. These con- 
trols essentially account for whether a given 
private school served the grade level of a 
given public school student and whether it 
actually accepted vouchers during the spe- 
cific year observed. That is, these controls 
move the analysis from “there’s a private 
school that accepted vouchers at some 
point” to “there’s a private school that ac- 
cepted vouchers for this student’s grade 
level in the year that we’re analyzing.” 

Coefficients for the combined test-score data 
are given in the report’s Table A. Greene 
and Marsh choose to interpret the analysis 
employing no controls, though the con- 
trolled coefficients are given in the bottom 
segment of the table. The issue of control is 
important, in part because — as explained 
later — the controls greatly reduce effect size 
(and statistical significance), and in part 
because it is not clear why FLR by itself was 
omitted from the analysis as a control vari- 
able (or covariate) at the student level. 



Though not without construct validity is- 
sues, FRL is commonly used as an indicator 
of opportunity to learn (OTL), given that 
students eligible for free lunch generally 
have less access to educational resources. 

Keep in mind that FRL is currently included 
in the model, but only as an interaction with 
N. Though it would seem plausible to in- 
clude both FRL and FRL interactions in the 
same statistical model, this action would 
likely result in estimation problems that 
could be metaphorically explained as having 
two star quarterbacks on the same team: 
team performance may become erratic, and 
consequently it may be hard for the coach to 

o 

judge the merits of either quarterback. 
There is also the conceptual problem that 
FRL is required to play offense (as a meas- 
ure of voucher eligibility) and defense (to 
control for OTL). Consequently, the interac- 
tion FRL*N could also be interpreted as a 
positive effect of voucher options for stu- 
dents with lower OTL. This is not necessar- 
ily the same thing as competition. 

The statistical modeling aspect of this report 
could be more fully developed with a multi- 
level approach in which students are nested 
within schools. Multilevel modeling tech- 
niques are useful and often statistically nec- 
essary when observations have this struc- 
ture. 9 From this perspective, the voucher- 
options variable is an interaction between 
the salience of competition at the student 
level and the availability of vouchers at the 
school level. It would be difficult, in my 
opinion, to construct a convincing argument 
for deleting FRL prior to examining empiri- 
cal results. 

In sum, the model specification in EMPCP 
is ultimately applied without including con- 
trols and does not include FRL as a main 
effect. The multilevel approach might lead 
to a more convincing account of model 
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specification. 10 Standard errors in the multi- 
level framework are also likely to be more 
accurate than those obtained with OLS given 
nested educational data. 11 

V. Review of the Validity of the 
Findings and Conclusions 

The report highlights uncontrolled results, 
and annualized effect sizes are not calcu- 
lated. In general, the controlled coefficients 
(only for grade and year, since FRL was not 
included) in Table A are about one fifth the 
size of the uncontrolled coefficients. Trans- 
lating this factor into Table F for the com- 
bined sample would result in effect sizes of 
.020, .016, and .020 for Language Arts, 
Mathematics, and Reading, respectively. If 
these gains were annualized over the seven 
years (the span of the data collection), it is 
clear they are very modest in absolute size. 

This observation should not be read as an 
attempt to minimize the importance of the 
statistically significant results reported in 
this study. Even small gains with crude 
measures of competition may signal more 
powerful underlying effects. However, it is 
critical in any policy research to distinguish 
clearly between statistically significant esti- 
mates and practically significant results. 

The report focuses on system-wide competi- 
tion (under the Table A column “Whole 
City”). According to Greene and Marsh (p. 7): 

It may be difficult to visualize ex- 
actly how MPS more effectively 
serves students with more voucher 
options given the difficulty of school 
officials to make fine distinctions be- 
tween what is provided to each stu- 
dent. But school officials do not have 
to make fine distinctions for them to 
be more attentive to students with 
more options. 



The authors appear to believe that if positive 
macro-level effects can be estimated relia- 
bly, then explanation or interpretation of the 
causal mechanism is secondary. A great deal 
of faith is thereby required in standard linear 
regression without controls as well as sub- 
stantial forbearance for the puzzling finding 
that competition effects were unrelated to 
voucher-school proximity. The downside to 
the macro approach to evidence-based rea- 
soning is the potential frailty of the assump- 
tions of model specification and of construct 
validity for key independent variables. 
However, the suitability of these assump- 
tions is most appropriately a topic for future 
studies, because the competition effects es- 
timated for Milwaukee public school stu- 
dents are relatively small, and a good deal 
more robustness checking is required prior 
to interpreting the importance of the compe- 
tition estimates with confidence. 

Properly presented and understood, and set- 
ting aside the specification concerns raised 
earlier, the results in EMPCP are consistent 
with the conclusion of Belfield and Levin (p. 
297): “the effects of competition on educa- 
tional outcomes appear to be substantially 
modest.” The policy implications are thus 
appropriately refocused on whether there are 
(1) better programmatic competitors than 
voucher systems for increasing student 
achievement, and (2) whether there are unin- 
tended consequences requiring evaluation. 
For example, in a recent policy brief, Arsen 
& Ni wrote: 

The research surveyed here suggests, 
rather than conclusively establishes, 
that competition from vouchers and 
charter schools is no more beneficial 
for TPS [traditional public school] 
performance than competition from 
nearby private or public schools in 
environments with no choice policy. 
Indeed, Belfield and Levin’s review 
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of studies of these traditional forms 
of school choice shows a higher pro- 
portion of findings indicating statis- 
tically significant positive effects on 
TPS outcomes than is evident among 
existing studies of voucher and char- 

1 9 

ter school competition. 

Obviously, the issue of whether competition is 
most effectively implemented with voucher- 
versus non-voucher schools is an important 
question that involves economic analysis and, 
more importantly, careful consideration of 
alternative value systems regarding public 
education and long-term consequences. 

VII. Usefulness of the Report 
for Guidance of Policy 
and Practice 

Before the results of this study can effec- 
tively inform the decision of whether to ex- 
pand or reduce the size of the voucher pro- 
gram in Milwaukee, a number of issues need 
resolution. Most importantly, the use of un- 
controlled estimates requires justification. 
After all, Greene and Marsh argue that sta- 
tistical control is important: “we believe 
controlling for year effectively ‘de-trends’ 
test results without comparable loss of in- 
formation” (p. 8). Why then, were uncon- 
trolled estimates preferred? 

Second, a more comprehensive approach to 
statistical modeling would allow important 



distinctions between student and macro-level 
effects. Third, an annualized metric should be 
used to compare student achievement gains 
spurred by competition with gains from other 
efficient educational interventions. If the au- 
thors would counter that it is not important to 
report annualized gains, then a well-reasoned 
argument should be offered. This would clar- 
ify how the effects of competition can be 
compared with other potential interventions 
(or competition studies in other communities) 
for increasing student achievement. 

Greene and Marsh (p. 10) may have antici- 
pated such reactions with the caveat “No 
single study of a single program can be de- 
finitive...” However, the ensuing sentence 
in the closing paragraph of EMPCP is prob- 
lematic: “but this research contributes to the 
finding that expanded choice and competi- 
tion improve the academic performance of 
students who remain in traditional public 
schools.” I remain cautious, if not a bit skep- 
tical, of the latter claim, and I would await 
further rationale and corroborating evidence 
along the lines suggested in this review. 

To spur further research, it would be most 
helpful if the authors would make their data 
set freely available for secondary analyses. 
This is a great opportunity to challenge other 
researchers to explore the data from differ- 
ent perspectives, potentially leading to more 
robust estimates and interpretations of the 
effects of school competition. 
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(3) b u = b t + y n Voucher + v u . 
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